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DETAILED ACTION 



Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1 .17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.1 14, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.1 14. Applicant's submission filed on 4/2/08 has been entered. 



Claim Rejections - 35 USC § 102 
2. The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the mvention was known or used by others in this country, or patented or described in a printed pubUcation in this 
or a foreign country, before the invention thereof by the applicant for a patent. 

(b) the invention was patented or described in a printed publication in this or a foreign country or in pubUc use or on 
sale in this country, more than one year prior to the date of appUcation for patent in the United States. 

(e) the invention was described in (1) an application for patent, published imder section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 35 1(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 



3. Claims 1, 2, 3/1 are rejected under 35 U.S.C. 102(b) as being anticipated by Michael et al 



(US 6,421,458 62). 
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Regarding claim 1, Michael discloses an image processing device for determining three- 
dimensional position and/or orientation of an object, comprising: 

image data capturing means for capturing image data containing an image of the object 
(see figure la, numeral 100, col. 4, lines 31-45, imaging system 100); 

reference model pattern creating means for creating a reference model pattern based on 
image data of a reference object with a three-dimensional reference orientation relative to said 
image capturing means captured by said image capturing means, said reference object having a 
shape substantially identical to that of the object (see fig. 2a, numeral 204, col. 4, lines 45-58; an 
alignment model image for the object is determined where the alignment model image 1 14 is 
selected from one of or part of the captured training images, the alignment model may comprise 
the entire training image itself or a portion of the training image containing interest features of 
the object); 

transformation means for performing two-dimensional and geometrical transformation on 
the created model pattern using a plurality of parameter sets to generate a transformed model 
pattern representing an image of the object with three-dimensional orientation different from the 
reference orientation (see fig. 2a, numeral 210, col. 4, lines 59-67, col. 5, lines 1-65; alignment 
model image 1 14 and each training image 101 are presented to an alignment system 106 for 
determining the affine pose 107 of each training image with respect to the alignment model 
image and uses the affine pose 107 comprises a set of parameters which describe how the 
training image can be transformed mathematically so as to align the training image the alignment 
model image, each filtered training image 109 and its corresponding affine pose parameters 107 
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are applied to the General Affine Transform 1 10 to generate transformed training images 111, 
following transformation, a template image 1 13 is computed in step 212); 

pattern matching means for performing a pattern matching of the image data of the object 
captured by said image capturing means with the transformed model pattern (see fig. 2b, numeral 
234, col. 6, lines 35-67, col. 7, lines 1-8; generate a transformed image 311 which align 
substantially with the template images 1 13); 

selecting means for repeatedly performing the generation of a transformed model pattern 
(see fig. 2a, numeral 216; col. 6, lines 8-20; a determination is made as to whether training is 
complete, if not additional training images may be captured) and the pattern matching of the 
image data of the object with the transformed model pattem to thereby select one of the 
transformed model patterns in conformity with the image data of the object, and obtain 
information on a position of the image of the object in the image data (see fig. 2b, numeral 234, 
col. 3, lines 5-10, col. 6, lines 35-67, col. 7, lines 1-8; process of transforming the filtered 
training and run-time images implies that the run-time process is executed more than once; 
generate a transformed image 311 which align substantially with the template images 113); 

means for obtaining information on a position of the image of the object in accordance 
with the selected one of the transformed model pattems in the image data (see col. 3, lines 5-10, 
col. 6, lines 35-67, col. 7, lines 1-8; statistics on the area, position, and orientation of the labeled 
regions are computed); 

means for obtaining information on the three-dimensional orientation of the object based 
on one of the parameter sets used for generating the selected one of the transformed model 
pattems (see col. 4, lines 7-18, col. 4, lines 66-67; col. 5, lines 1-17; affine parameters apply to 
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six degrees of freedom to compensate for image scale, shear, rotation, skew, and translation 
assuming a two-dimensional image of a three-dimensional object); and 

determining means for determining three-dimensional position and/or orientation of the 
object based on the information on the position of the image of the object in the image data and 
information on the three-dimensional orientation of the object (see col. 3, lines 5-10, col. 6, lines 
35-67, col. 7, lines 1-8; statistics on the area, position, and orientation of the labeled regions are 
computed). 

Regarding claim 2, Michael discloses an image processing device for determining three- 
dimensional position and/or orientation of an object, comprising: 

image data capturing means for capturing image data containing an image of the object 
(see figure la, numeral 100, col. 4, lines 31-45, imaging system 100); 

reference model pattern creating means for creating a reference model pattern based on 
image data of a reference object with a three-dimensional reference orientation relative to said 
image capturing means captured by said image capturing means, said reference object having a 
shape substantially identical to that of the object (see fig. 2a, numeral 204, col. 4, lines 45-58; an 
alignment model image for the object is determined where the alignment model image 1 14 is 
selected from one of or part of the captured training images, the alignment model may comprise 
the entire training image itself or a portion of the training image containing interest features of 
the object); 

transformation means for performing two-dimensional and geometrical transformation on 
the created model pattern using a plurality of parameter sets to generate a transformed model 
pattern representing an image of the object with three-dimensional orientation different from the 
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reference orientation (see fig. 2a, numeral 210, col. 4, lines 59-67, col. 5, lines 1-65; alignment 
model image 114 and each training image 101 are presented to an alignment system 106 for 
determining the affine pose 107 of each training image with respect to the alignment model 
image and uses the affine pose 107 comprises a set of parameters which describe how the 
training image can be transformed mathematically so as to align the training image the alignment 
model image, each filtered training image 109 and its corresponding affine pose parameters 107 
are applied to the General Affine Transform 1 10 to generate transformed training images 111, 
following transformation, a template image 1 13 is computed in step 212); 

storage means for storing the plurality of transformed model patterns and the parameter 
sets used in generating the respective transformed model patterns to be associated therewith (see 
figure la, numeral 1 12, col. 5, lines 39-53; transformed training images are preferably stored in a 
pair of accumulators 1 12); 

pattern matching means for performing pattern matching of the image data of the object 
captured by said image captmng means with the plurality of transformed model patterns to 
thereby select one of the transformed model patterns in conformity with the image data of the 
object (see fig. 2b, numeral 234, col. 6, lines 35-67, col. 7, lines 1-8; generate a transformed 
image 31 1 which align substantially with the template images 113); and 

means for obtaining information on a position of the image of the object in accordance 
with the selected one of the transformed model pattems in the image data (see col. 3, lines 5-10, 
col. 6, lines 35-67, col. 7, lines 1-8; statistics on the area, position, and orientation of the labeled 
regions are computed); 
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means for obtaining information on the three-dimensional orientation of the object based 
on one of the parameter sets used for generating the selected one of the transformed model 
patterns (see col. 4, lines 7-18, col. 4, lines 66-67; col. 5, lines 1-17; afifine parameters apply to 
six degrees of freedom to compensate for image scale, shear, rotation, skew, and translation 
assuming a two-dimensional image of a three-dimensional object); and 

determining means for determining three-dimensional position and/or orientation of the 
object based on the information on the position of the image of the object in the image data and 
information on the three-dimensional orientation of the object (see col. 3, lines 5-10, col. 6, lines 
35-67, col. 7, lines 1-8; statistics on the area, position, and orientation of the labeled regions are 
computed). 

Regarding claim 3/1, Michael teaches a two-dimensional and geometrical transformation 
of an affine transformation (see col. 4, lines 66-67; col. 5, lines 1-17), and said image processing 

device further comprises additional measuring means for obtaining a sign of inclination of the 
object with respect to said image capturing means (see col. 4, lines 66-67; col. 5, lines 1-17). 

Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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5. Claims 7 and 8 are rejected under 35 U.S. C. 103(a) as being unpatentable over the 
combination of Michael et al (US 6,421,458 B2) in view of Watanabe et al (EP 1043689 A2). 

Regarding claims 7, 8, Michael discloses all elements as mentioned above in claim 1 . 
Michael does not disclose a storage means storing an operating orientation of the robot relative 
to the object or storing an operating orientation and an operating position of the robot relative to 
the object; and robot control means for determining an operating orientation of the robot or the 
operating orientation and an operating position of the robot based on the determined three- 
dimensional position and/or orientation of the object; and an image capturing means mounted on 
the robot. 

Watanabe, in the same field of endeavor, teaches a storage means storing an operating 
orientation of the robot relative to the object or storing an operating orientation and an operating 
position of the robot relative to the object ("when a picking-up command is inputted .... three 
dimensional position/posture of the camera 20 on the world coordinate system at this image 
capturing position is outputted to the image processing apparatus 30"; Michael: paragraph 
[0040]); and robot control means for determining an operating orientation of the robot or the 
operating orientation and an operating position of the robot based on the determined three- 
dimensional position and/or orientation of the object ("robot controller 10 operates the robot to 
perform a pick-up operation .... based on the three-dimensional position/posture of the 
workpiece W (Step 210)"; Michael: paragraph [0044]); and an image capturing means mounted 
on the robot (Michael: figure 1, numeral 20). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Michael reference, to be incorporated into a robot system as suggested by 
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Watanabe, to enhance inspection process by "demanding relatively less on system storage, and 
improv[ing] system speed and accuracy" (Michael: col. 2, lines 5-12). 

6. Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over the combination 
of Michael et al (US 6,421,458 B2) with Maeda et al (US 2003/0161537 Al), and further in view 
of Inoue (US 2003/0161504 Al). 

Regarding claim 4, Michael discloses all elements as mentioned above in claim 3. 
Michael does not teach dividing of a model pattern into at least two partial model pattems which 
are subject to the affine transformation to generate transformed partial model pattems, and 
pattern matching of the image data of the object with the transformed partial model pattems to 
determine most conformable sizes, and determines the sign of the inclination based on 
comparison of the sizes of the conformable partial model pattems with each other. 

Maeda teaches a pattem matching of the image data of the object with the transformed 
model pattems to determine most conformable sizes, and determines the sign of the inclination 
based on comparison of the sizes of the conformable model pattems with each other (Maeda: 
figure 1, numeral 32). 

It would have been obvious at the time the invention was made to one of ordinary skill in 

the art to modify the Michael reference to determine the sign of the inclination based on 
comparison of the sizes of the conformable partial model pattems with each other as suggested 
by Maeda, to allow the imaging device to accurately identify "a three-dimensional object using 
images obtained by photographing the object in the various directions" (Maeda: paragraph 
[0004]). 
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The Michael with Maeda combination as applied above does not teach dividing of a 
model pattern into at least two partial model patterns which are subject to the affine 
transformation to generate transformed partial model patterns. 

Inoue teaches dividing of a model pattern into at least two partial model patterns (Inoue: 
figure 2, numeral 302, 303), which are subject to the affine transformation ("affine 
transformation"; Inoue: paragraph [0107]) to generate transformed partial model patterns. 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Michael with Maeda combination as applied above, to utilize partial model 
patterns as suggested by Inoue, to further "correctly classify the input image regardless of a 
fluctuation in illumination, and a state of occlusion" (Inoue: paragraph [0012]). 
7. Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over Michael et al (US 
6,421,458 B2) in view of Okisu et al (US 6,806,903 Bl). 

Regarding claim 5, Michael discloses all elements as mentioned above in claim 3. 
Michael does not teach a measurement of distances from a displacement sensor separately 
provided in the vicinity of said image capturing means to at least two points on the object using 
the displacement sensor, and determines the sign of the inclination based on comparison of the 
measured distances. 

Okisu teaches a measurement of distances from a displacement sensor separately 
provided in the vicinity of said image capturing means to at least two points on the object using 
the displacement sensor, and determines the sign of the inclination based on comparison of the 
measured distances ("measure distance to two separate portions of an object from the electronic 



Application/Control Number: 1 0/807,259 Page 1 1 

Art Unit: 2624 

camera, and calculate an angle of inclination 0 based on measure two distances"; Okisu: col. 19, 
lines 54-57). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Michael reference to determine the sign of the inclination based on 
comparison of the measured distances as suggested by Okisu, to allow the image processing 
device to properly focus with high degree of accuracy on the targeted object for image capturing. 
8. Claim 6 is rejected under 35 U.S.C. 103(a) as being unpatentable over Michael et al (US 
6,421,458 B2) with Watanabe et al (EP 1043689 A2), and further in view of Maeda et al (US 
2003/0161537 Al). 

Regarding claim 6, Michael discloses all elements as mentioned above in claim 3. 
Michael does not teach additional pattern matching of image data of the object captured after 
said image data capturing means is slightly moved or inclined and determining the sign of the 
inclination based on judgment whether an inclination of image of the object becomes larger or 
smaller than the selected one of the transformed model patterns. 

Watanabe, in the same field of endeavor, teaches an additional pattern matching of image 
data of the object captured after said image data capturing means is slightly moved or inclined 
("camera may be moved parallelly in accordance with the position of the workpiece in the field 
of view of the camera"; Watanabe: paragraph: [0048]). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Michael reference to move the image data capturing means as suggested by 
Watanabe, to provide more training images to the classifier in order for a more robust collection 
of data to enhance the classifier for increased reliability of the system. 
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Maeda teaches additional pattern matching of image data of the object captured after said 
image data capturing means is slightly moved or inclined (see fig. 4) and determining the sign of 
the inclination based on judgment whether an inclination of image of the object becomes larger 
or smaller than the selected one of the transformed model pattems (Maeda: figure 1, numeral 
32). 

It would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify the Michael with Watanabe combination to determine the sign of the inclination 
based on judgment whether an inclination of image of the object becomes larger or smaller than 
the selected one of the transformed model pattems as suggested by Maeda, to allow the imaging 
device to accurately identify "a three-dimensional object using images obtained by 
photographing the object in the various directions" (Maeda: paragraph [0004]). 



Response to Arguments 
9. Applicant's arguments with respect to claims 1-3, 7, 8 have been considered but are moot 
in view of the new ground(s) of rejection. 

Regarding claims 4-6, applicant argues that the claims are patentable because of the 

dependency from claim 1 . This argument is not considered persuasive since a new grounds of 
rejection have been made and can be seen above in the rejection of claim 1 . 
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Conclusion 

10. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to EDWARD PARK whose telephone number is (571)270-1576. 
The examiner can normally be reached on M-F 10:30 - 20:00, (EST). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Vikkram Bali can be reached on (571) 272-7415. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an apphcation may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Edward Park 
Examiner 
Art Unit 2624 

/Edward Park/ 
Examiner, Art Unit 2624 



/Vikkram Bali/ 

Supervisory Patent Examiner, Art Unit 2624 



